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ABSTBiCT 
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■ethod for user evaluation of purchased scientific and technical 
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■atriz of paraeeters and variables for the user to apply in selecting 
between coepeting services of this type. The eatriz approach eas 
selected because of the varying ieportance of quantitative, 
qualitative, and business judgement factors to different types of 
decision eaker. (Author/P?) 
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TECHNICAL REPORT 



I. Objectives and Deflnlttons 



This contract is part of the NSF/OSIS program on Lnproved 
Dissemination and Use of Scientific and Technical Information^ and ms 
awarded under the geaeric title. The' specif Ic subject matter to be covert 
is the preparation of a "generalized method for the user evaluation of 
purchased Information services^" and we-pcopose to use this as a descriptive - 
sub-title In all reports. Special points of emphasis will include the-pro7^ ! 
bl ems and methodology of handling heavily biased information^ which is 
characteristic of the literature In this field and also of the highly . 
divergent opinions pf. different users or the same services. ^ This statement 
of objectives was discussed in initial conferences with the Technical 
Monitor and strategy preview sessions with Internal advisors and consultants^ 
during the first months and confirmed in further discussions with Vr. Joel 
Goldhar for NSF. ' ' - - . ' > 

The progrant is set up as the first year pf. an anticipated two* / 
year project. The distinction between^ tlie first and' second years Is. based 
on the degree of soph^istication of the information systems considered.- The/^~ 
first year is directed. at purchased, services^ oftea.of"hlgh■quality^:wl^lch^: -:; 
provide information without interpretive evaluation or analysis: this is 
stated. &q>licity» for example^ by FINO/SVF. The object to develop a matrix - 
of parameters and variables for the user to apply^ in selecting between c<m^ 
petlng services of this type. The object for the second yjear is to extend 
this approach to the evaluation of the much more coa^lex operation of infor-- , 
ma^on analysis centers* These nay be ccnnpletely in-hause» or If not^ they 
Involve direct technical interaction in-house for their effective use. ~ 

The^^definitiqns of Jterms to be used in/this program'.preseot -semantic 
problems which must be' approached by arbitrary choice^ ^after due consideoratlon 
since there ;f.s no answer on any other 'basis: every word availabla has been 
given many different and often contradictory meaninga by different experts;, at- 
dlfferent^ times. The definition of ^^servlces" to be considerodj herein. 
Includes processed -inf onnatlon', Vto" exclu4^\'oiigiul^ equivtiileEtt,. , 

sources^ It^m^y jjiclude^ three major categorl^^: - (a) abetractrand indexing . 
services^ whiich are^publi^hed and sold "^i . tools .for "self ^isia^:^^^^ 
searchingfserylceSy which provide no analyses of^th^^'out^^ Ijifotmation^ahd-^'-' 
no reinterpretation:.pf -Vthe ^question:duritigrtfhe .cc^^ th<^ -search»*'and'^^:; 
(c) information analysis services^ which include evaluation and feedback 
during the search. The first year program will examine the bases of selec-* 
tion used in selecting between purchased information services for internal 
use» category (a) above^ and ^tend this to consider the same ^d other 
variables in user selections between competing services in simple searching* 
category (b). Both of these aspects will be based on a case history approach 
to Contractor's experience^ in ER&E and the information centers of affiliates. 
This has proceeded far enough to start extending the same approach into field 
interviews with other users. 



Xhe present contract^ considered as Phase I of a two-year project^ 
will consider the extent: to which positive and negative selection principles 
derived from internal <:zxperience can be generalized^ to apply to external 
purchased searching services. Phase 1I» for the second year» is contemplated 
as a logical extension of the matrix of variables developed in Phase 1» to 
examine additional variables >r parameters based on user experience with . 
more sophisticated systems. 

# 

7he definition of **user" also has many different aspects; the 
essential element for the purposes of this project is the person who makes 
the selection of which service to employ^ when there is a choice to be made, 
^his covers a complete range: the user may "be a searcher^ a manager^ or an 
information analyst; the same person may be both user and customer ^ the 
one who asked for inCormatioa " but this is not the usual case. In by far 
the most caseSy the user is collecting information for someone else. The 
customer may get all the original literature and iadesc materials in a simple 
search^ or he may make selections from an index and request, only specific 
items. Systems where the user does the analysis and .confers vlth tfae customer 
whjCle completing the search^ or where the user does all tlie analysis aoif sup- 
plies only a digested product^ are not included directly In Phase I. The 
matrix of variables developed will be open^ended^ with this objective in . . ' 
mind. ,.It must be recognized that dittermt principles will apply to the same 
service for original purchase rjid for daily use» or for use by different 
.people" for. different purposes. ' - 

Case History Approach 

Preliminary interviews have been conducted with users from three . 
different groups » incl\iding senior information analysts vho are advisors * 
to th^. projcxty Information system managers who were consulted at the MSF 
conference at Hennlker^ N.U. In August^ and skilled information searches 
from Internal staff. , . . ' ^ 

. Two .case histories have been ideatlfied based on internal exper-* ' 
ience» in the purchase; of major information services. .For the time being i 
.in this report^ these and all other specific examples will be left unidentified. { 
This decision ia subject to. further discussion and review^ for later reports. [ 
The first example refers to an 'expensive^ indexing service, which ha<l been " . V 
'examine^^bn a trial basis but left untouched on the shelves^ because it was 
.unfamiliar, to potential users- and required some training to be useful to 
them. . . ^ :■ - ' ■ ■ ■ 

The initial reaction was negative^ but "trade information" over 
ensuing years indicated identifiable unique advantages and satisfactory 
experience elsewhere. On a subsequent visit to the vendor an improved 
indexing system was offered^ based on additional years of experience^ but 
requiring a substantial initial purchase cost. By happy coincidence^ 
upon returning from this trip» the manager concerned was informed by 
Accounting that an unanticipated balance of $X»000 was left in a capital 
reserve account^ and the iiew service was purchased at once. Six weeks 
later Accounting -called again to say they had made a mistake^ but by that 
time the new service was already in use» with enough experience to easily 
justify its continued cost. Analysis of this story reveals several factors: 
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the "energy barrier" to the initial purchase was apparently too high* and 
the company might have benefitted sooner by allowing the manager more 
"risk capital" for new experiments. Along a different line» this and similar 
e:cperiences In the evaluation of research innovations suggest that a 
successful experimerit is frequently accompanied by what appear to be 
lucky breaks^ but are in fact a basis for serendipity or maaageiiieat hunches 
based on experience. The exact converse of this is that in experiments 
which turn out to give negative answers* or failures* there appear to be 
many unexpected gremlins or quirks which there was no logical reason to 
anticipate. The frequency of such events suggests that intuition or. the . 
managerVs hunch is .In fact an essential' dimension to consider in .the decision 
making process . / , . : . ■ . - ^ ■■■ ^- ' ■ 

A second case history identified was somewhat analogous. In this 
case a major service offered in several sections had been' purchased (in part) 
by an affiliate library^ based on Information in sales brochures. It had 
piled up unused on the shelves -because 'of the considerable clerical time 
required to set it up and get started* and to interfile additional sections 
as. received to keep .it. current. 'A major search request was received at 
Linden which could justify. the time to convert-.this file to usable form* and 
"It vas boxed up and shipped off promptly by the affiliate^ vtiic\ was glad 
to clear. its shelves; The result again was a complete succesr^^ and led to. 
expanded'usefulness based on further experiences. These ca'^us. identify 
major variables l^ln .the .selection .process^'including maintenance or keep* ^. 
ready time! as well as Initial and contlntilng costs... . - 

The next rtep was to take the variables thus identified and con-* 
sider them» with others^ as selection principles for another us^. The 
question chosen was. to construct a list of internally purchased Indexing 
and abstracting services which are considered as "unique^^* not replaceable 
by anything else. This vas ^cpanded Into the table, attached as List.l* 
-to consider factors which are useful In^selecting betweea * services . Please' 
note that ..this. is. a yery preliminary llst» based on approximate information 
and opinion^ 'and subject to correction and refinements on further considera* 
tion. Its chief intended value at this time is not its content but the type 
of questiort whichr proves^useful as a column' Heading* - to serve as a disr- 
crlminant factor ift the'choice between con^etlng sefvices;". No attempt has ' '* 
beert made to^^i^acCd-bbvlous headings such as "special field (s) .covered*'' - / 
where "imicIf'niOT Infbrmatlbn" is available.""" " ^ 'r^ ■ " "^^ ' " 

' ' ^r^Thi.S::.list 1^ not'arranged at this time in. any particular prder» 
except that. die fist six entries were considered moiit valuable for most"'"-^ ''" 
users. In an: attempt -to generalize why this choice was made* it appears 
that these are services where the same amount of basic training* experience* 
and facilities makes it possible to answer more different types of questions 
than a competing service* which is rated less valuable. A different con- 
clusion based on the same data could be that because these services are 
used more of ten » the users became more skilled in them and find them 
easier to use. This is not necessarily a different conclusion^ however^ 
since both may be related to the observation that this service is more 
flexible^ for more users. 



List 1 has been backed up and confirmed by a preliminary Inter-^ 
vieur at a nearby pharmaceutical firm. They recomiaend adding to the list . 
BXOSXS and >IEDLAKS/M£DLINE» with the notation that both of these require 
a major Investment of Initial training tlme» and regular continued practice 
to maintain the minimum skills necessary for their effective use. Another 
information analyst suggests Uniterm Index as an important addition to 
List 1. Further discussion of useful column headings has identified. 

timeliness*^ of the service in terms of the ranges of time required^ 
rather than an average or expected time» to get items out of the current 
literature entered into the system. 

A third case history approach was considered next» as summarized 
in the attached List 2. This is rated as a largely unsuccessful attempt 
to categorize ftome 75 of the Information^ sources/services used most often 
by skilled searchers at BR&E. The first question asked here» and the basis ' 
of the column headings^ was to consider those information sources wUlch 
have been or might be called by telephone (or by letter request) > to ask 
for further information on what had appeared in printed copy^* A fourth 
column was added for Additional items considered by" this' worker as a 
significant information resource^ and additional entries in the first column 
for services which do not depecid on printed copy. Se^reral problems with' ^ 
this list appear from the marginal Indexing flags entered before, specific . 
items: "1^* refers to the first six entries on^Llst 1 and *V*.the second 
eight entries there; the mark before ■dlfferent.,ltems identified which , 
ones this same worker selected 'as most useful as a searching tool » regard** 
less of the discriminant question asked for the column headings* It is 
Immediately apparent that any such list will vary widely from worker to 
worker » or for the same worker depending on what question. is asked. A 
possible approach for the further analysis of these data is to consider 
what types of questions get the widest fluctuations . from one worker, or 
one use to another^ and what questions are most likely to give a better 
matching of answers.. List 1 was probably .a good question on thla basls^ 
and List 2 was not. Additional questions of this' type will 'be identified 
and examined further. . \ " ' . ^ 

- Stated differently^ List 2 may be examined aa a basis for dia- ' 
agreements^ where Llst^.l may provide bases for agreements* This approacH 
is analc^ouQ , to Inputs received from Eldon' Sweeny on .the yaXue of negc^tive 
data> in a para^el'project for NSFon "A Study of Obstacles .to Innovation 
in the S.T.I. Services Industry." this contains some very useful data on 
the significance of specific negative factors. ^ The present project differs 
significantly from Sweezy in one respect^ in being directly opposed to the. 
basic principle of anonymity in the Delphi approach. Our approach is to 
examine the nature and reasons for individual bias as carefully as possible^ 
and report them anonymously if desired^ rather than concealing them through-* 
out» so as to develop a fuller understanding of the decision-making process. 
This approach may be of special Interest to some of the heavily over'- 
interviewed users*' considered in various parallel projects^ and suggestions 
as to how this might be developed may be available through NSF. 
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A dimensionally different approach to thp question of how selections 
arc made between competing services appeared during discussions with attendees 
at the NSF-sponsored Engineering Joint Foundation Conference in Henniker^ N.H. 
from August 10 to 15. This question was explored philosophically with a 
number of people who might be categorized as ^'user/managers.*' A common 
element In their reply was *'for a new service or a hew use» I would always 
try to find a manager whose judgment I trust, iTho had used this or some- 
thing like and ask his opinion." The basis of such business judgment 
is not a quantifiable^ numerical factor^ and it Is often a go/no-*go decision 
rather than one. that can be ranked by subjective degrees. . - 

Dlmegslons in the Selection Process ' \.\ \ \..'\ / / 

The matrix, approach Is being considered here for "the selection of 
variables in a decialonnnaking process^ The validity of this approach 
depends upon finding different categories or dimensions for groups of 
variables In the same group and different from another group of variablea 
in a different dimension. - " 

Systems' for the "evaluation of research programs", have bean pro^ ' - . . ^ 
posed which started with a dozen or more auch groups of factors, or dlmenaions* 
but greater success has been achieved when the number of dimensions Is _ 
gi.eatly reduced* preferably to three. If such d±:aensions can be found* the \- ^ — ■ 
problem of overlap between equivalent or .dependent factors In different \ ■ ^;.v 
groups may be much simplified. ".. ^ . / - . " " / " 

The dimensional analysis scheme being currently considered is sum- . 
marlzed In Table 1. ^ Three dimensions are propoaed: quantifiable factora* which 
are heavily emphasized In the literature; subjective factors* which are fre* 
quentlY handled by "ranking" techniques; and factora In busineas judgment which. ?- : 
may have yes/no answers* neither quaptlflable nor. ranked. A j^ubtle correlation 
^Ich aeems.to justify ttiis. choice of dimensions Is that each of thca-appeala . ^ 
most to a different category -o£ users: .the vendor places 'heavy emphasis on 
cost or other quantifiable factors. He can advance .these as a sales agrument* 
or In the literature -on systems evaluations whlch-haa been written by the purveyors 
of new.systems. . This emphasis In the literature Is so strong, In fi\ct* that\. 
It frequently tries -to.^jleny .the validity jof ,ai^fa(ltpr which cannot. aom^ow be 
quantifiedj^^. Conclusions baaed cm cost alone mus^t"^be'accepted*with cautlo,'ci*-_^/ 
howaverj^.:slnce hlgK;Cost may b&.llidced dlrec^tly to. high quality aa in the , ^ 
co&t/benefits'^df anZexpensive {^ge of advertising. Factora' :ih the quantitative 
dimension may also. change markedly In relative: Importance at 'different levels _ 
of activity": ■ For initial use/ promptness of'servlce or ho*f much work it is to* ' 
use may be most significant, ^ereas relative costs become more Important for 
the second look. 

Major company consumers of Information* who are considered as - 
customers rattier than **users" under the arbitrary definitions given above, 
reached an entirely different conclusion. Three^ different men of this 
type agreed that they pay little or no attention to coat/benefit factors* 
once an available service has been listed by managment experts as reliable: 
any service that provides the Information they need is worth what it coats* 
and it will not stay in business if it costs too much. Their choice between ' 
services is based instea<? on subjective factors, summarized by the question 
''is it a pleasure to use?" 



Almost any user of Information services has access to the advice 
ot some manager^ whose experience he can consult. The user/managers* 
factors of business judgment are harder to define^ but they add up to 
questions of reliability and experience. It is the vendor /manager who 
is concerned with sales and production costs» and quality standards ^ich 
can be set as specific targets. Thus, Kathleen Bingham of FIND/SVP saya 
**tlme is all ve have to sell^" and places the highest value on proiopt 
service, repeat business as a measure of reputation^ and depth of experience 
as measured by "areas in which you go to primary sources for information** 
rather than to handbooks, reviews, or encyclopedias. Irene Farkas points 
out that many sei^ices have no quality factor at all» only to get the Infor- 
mation out: some of the' simplest quality^ factors 'are in 'the use^of **koown 
sources of data»** or the ability to give the aame question to different 
searchers In the system^ and come up with the same product. Theae judgments 
are'basedy of course, .on, feed-back from the eiq[)erlence of customers and 
vendors^ as well as from other managers, and all of these dlmenaions 
interact continually In the selection process. Snap Judgmenta based on a 
single experience are recognised as dangerous » to be regularly checked by: . 
ai^ independent or aecond look for permanent values;, V- * ^ \ ^ 

Other combinations of 'dimensions are entirely possible and 'these - 
are being considered for whatever advantages they of fer» as they appear.. 

Current Program 

The comments^ case histories, and dimensional analysis above are 
being correlated Into a questionnaire or diacussion guide^ to be used in 
hand for further interviews with affiliates and In the field. Ve do .not 
now expect to use any questionnaires apart "^trom Interviews. 

- - < A large body of useful data .and correlations is available from' 

parallel and previous projects sponsored -by NSF, and the consideration of 
-this- material" Is being -started; -^^^The Trojeci 'Monifcor^haa^i^ 
gestions of specific individuals , In some of these projects 'who 'may be con^' 
suited for data on hand» or for Indications of people to be tnterviewe^d 
who might be particularly interested in our approach.' Any further sug- 
gestions along these linea-v*ill- be appreciated.- -:^ - - ^ v ' - 

-7 ^ - The review of the litwatur^ will- cdntihue^agAinst "the" liackground ' 
of the dimensional approach proposed, to augment and define the factors 
identifiedv^nd-^ethods^by'^jhich-'they interact. """"^ ^" * " " " '''''' 
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Services 



Ust l: ^ "Unique" Services 1„ Contractor ISxperience ' 



Derwent C.P.I. 
Plasdoc 
Chemdoc 

J^Chcm. Eng. 
I-Ilucleonics 
Refractories 

S.C.I. 

Prcdicasts 
CMA 

Funk-Scott 
Pred, Stat. 
Computer Files 



C*A. (hard copy) 
(microfilm text) 

Engg. Index 

Tape index 

API (on-line) 



Pollution Abs. 
Petroleum Abs. 
Applied Sci & 
Techn Index 
Fuel Abs* 
Env. Abs 

Nuclear Sci. Abs* 
Govt* Pubis. 
STAR 



Cost/yt. 
(-h Initial) 



Shelf 
Space/yr. 



mid* 
Training ■ 



Special FieldCs) Covered 

in 6 subject areas $30,000/yr* . 12' bulletins >2 daya mln* 



15,000/yr* basic, 6-* cards 
+ areas >3' microf * 



S*T.I* 

Chem* Markets 
Corps. & Industries 
S.I.c* bkgd: time series 
plant file 
all above & ACS 
Ind* Notes 

annual Indexes 
(right to ^ copy) 

**trans-dlscipltnary" 
in Engg* 

supplemental only 

betters aba & Indexing 
for petroleum, careful 
selection of item 

(buBiness, technology) 
applied, less depth 



$375 • 

425 ' 
.850 , 

150 ; J'-, : V 
$90/hr , . after 
hard copy - 

$3500 . . . 
$1750 ' 

■ ^ ■ 

$8j00 ■■ 
mo./aimtiltl 
■»$55/hrv 

pro-reta ijr 
company sis*- 



$120 



312 
100 
200 
, 75 
125 



, 24" 
36" 



■>.67yr. 
M8" 



3' lit. 
3 * . patant* 



12' 



dally, format 



10-15 -mln. 
2-4 hrs, 

1-2 daya 

'v/hr, 
1-2 hrs, 

> 2 days- 



10^1!» mln * 



er|c 



J 

■ Skills 



L 

1 



Maintenance: 
Keep*Ready Time 

2 hrs/vk 

8 hrs/vk 



Comments 



L none 

P able to browse weekly flUns 

. (interfiling) 

P or highly X 
trained clerks 



unique 
unique 

(since SEARCH quit) 
routine use» under contract 



shelf only 



P (>L) 



none 



Indexing requires care 
(archaic!) 



shelf only easier to search than hard 

copy 



different area» good for 
reviews* access to tech . 
shelf only trade ioumals 

(non* technical) 
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List 2, Interview 93 (8 Oct. 75) 
Info. Sources /Services Used by £R&E/CLIC 



T (frequent calls)* T Used 

l/API Abs. Air Polln Titles (do searches), 

1/ Chem. Abstracts / C&ES 

1/ Engineering Index CM Reporter 

Funk & Scott Index 
1/PREDlCASTS 
/ Chem Meek 

(McG/Hill Chem. Exbn. Dept.). 
Current Contents (I.S.I • pbcV SourceSj ^ 
1/ Dflrwent (PLASDOO, CHEMDOC;" etc); ' ^ 
a Environment Abs j 
L European Chem. Nev&\ 
a Petroleum Abstracts 
a Pollution Abs 
(Public Affairs Info Services) 
Science Abstracts 
1/ Sci. Citation Index 
/Statistical Abs. of U.S. i . 
Survey of Current Business (B.L.S) 
Thomas* Register^ 

Services Available 

ASIS (technical comms.) 

ASTM (technical comms.) 

AD Little (does searches) 

Battelle (does searches) t 
, Medlars on-line 
/ Lockheed on-line (33 bases) 

SDC on*line 

Pulp & Paper Inst. (etc. etc.) 



* Bases of ".cliissif icAtlon^ 

T . « Tel^ephone calls have been or cfould be made to ask questions 
L Lettersjhave been or could -be-vritten to ask questions 
/ { p 'Major i^6\ source for this .vorker,t regardless of above. 

i,:i^:fList;i ^^"n:.^^^ ■w; r: / y 

a AddltibMl eAtri^ for List l '(s6^ also Uniterm/ Index, BIOSI?) 
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Quid be X 



Current Programs 
NSF-RANN Energy Abs . ■ ' . 
! Science & Govt .Report,.;.'* 
Advanced Techno;.ogy Libraries 
Air Polln Abs , ■ ■ ' 
a/ Applied Sci. & TechnV Index 
, Automotive Englneeping' 
L British Teclm. Index.) ■ .. 
■■L Cheiaical Age .(Intl). { ■ 

Chcnj. Industry Jifotes, (to be on- 
.LChcm. Industrie ;(Hand«lsblatt)' 
' Energy Statistics' ;[ "y 
■• Tood Sci. & Technv Abs*: ■ ' I' 
, .Metals Abs Index.' \ ' ,, ; ' 
; Natl. Petr,. News' ' " '■■ ■*. 
' ■ RAPRA (British-rubberj 
' Water Polln Abs ; '.^Ov.,, ■ 
World Surfa<;e Cdatlhgs 'xbs • 



line)" 



Info resource: 

Analytical Abs. 
■ Applied Mechanics Reviews 
/ Aslib Booklist 
Building Science Abs 
Business Periodicals Index 
■■ Chemical Titles 
. V Cumulative Book Index 
Dissertation Abs Int'l. 
' Forthcoming Books 
; a Fuel Abs. & Current Titles 
I; . Gas Abstracts 

Highway Research Abs. 
;> Household & Personal Prods Industry 
/ Info Science Abs | 
:- Intl. Aerospace Abs. i 
; Library & Info. Sci. Abs. ! 
a/Monthly Catalogue U.S. Gov. Pubis. ! 
.a/ Nuclear Science Abs. j 
Rheology Abs. { 
■a/ Science & Tech. Aerospace Reports ^ 
World Meetings 
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Table 1 



Factors In Selection Between Competing Services 



Dlmensijons In the Selection Process: 
a) quantitative factors: 



numerical 
, (objective) 



b) . qualitative factors: 



rankings 

(includes subjective) 



c) business judgment:. 

* go/no go 

(includes intuitive) 
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cost factors (differ^ for first look and continued 
coverage of selected fields 
(no. of docum^ts in field) 
(% o£ available lit. in field), 
time factors tor deHvery 

(range^ as well as average) . . 
eKperleace in actual type of work involved 
t:(not just th^ field) * " - 

" skll!l to anticipate problems 
. --. eibpected re-defioltions of Q. 

fprmajt! of report - appearance-/'-* ■ ".^-V" 
■ - . , ■ ^ ■ — iorderiy presciitation ■ " " 

- - coovenieat access - vithiD" ' 
/ - report^ IndeKes^ arrangemeat* 
' x-indexes'^ h - 

flexibility of coverage * ability to vary 

.-.(+ select), deptti.pf . 

covenge; details 
" abUity to hl^lght- 
(selected items) by 
' / ■ ■ ... cloaenessi of-matcli to Q« 

responsiv<mess ciutvit priorities ' > : > 

; . ; ' ' ^'tolerate a^ " ^ ^ 

- viil suit Vour way .of doing thing 

feedback - communlcatlofi with or by user ^ ' 
.. ^.maans^ to redefine Q darliig search / 

^ " -Vr^suit^ility for browsl^ '; l. V-.:i.-i^. 



reputation ** reeonmendatlon of past^users - - 

-.reiiabiUty; unifortd.t^ quality* 

- confidentially 

- viability* will it last 
familiarity* to improve efficiency 

f actuality - if he says a thing is so* does he 
have a basis for it 



ctimslstracy - ri 



de^tudsbla bias 



